Skip to content

fix(transport): close upstream work once on client disconnect - #31

Merged
ALikesToCode merged 1 commit into
integrate/gateway-waves-1-2from
feat/request-cancellation
Oct 9, 2026
Merged

ALikesToCode merged 1 commit into
integrate/gateway-waves-1-2from
feat/request-cancellation

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Closes upstream work exactly once when a client disconnects, and records the result as uncertain rather than free or successful. This is a cleanup fix with no flag. Completed responses keep their exact bytes, status codes and headers.

  • Python:
    • iter_stream_content and iter_stream_lines now return closable iterators that own the upstream response from the start. A client disconnect, a read error, normal exhaustion or a repeated close releases the upstream once.
    • Unified dispatch binds each upstream response. A stream preflight rejection closes the same owner.
  • Worker: the roleplay transport moves its abort and close handling into worker/upstream-cancellation.mjs.
    • A client abort now reaches the provider fetch and cancels the upstream reader once.
    • It also stops candidate and origin fallback, so a disconnect never starts another provider attempt.
    • This brings worker/roleplay/transport.mjs under 1,000 lines.
  • Outcomes: after handoff, a disconnect is reported as ambiguous, with unknown usage and no replay permission. It is never reported as zero tokens or zero cost.
  • Behaviour change on errors: a failed socket read in an SSE stream now ends the stream with the error. It used to fall back to line iteration on the partly read response, which could repeat or garble data.

Details are in docs/request-cancellation.md, which also lists the paths not yet covered.

Integration

The Wave 3 integration PR connects these outcomes to request accounting. It also binds the provider-specific streaming generators in services/proxy_service.py, and passes the request signal to other native Worker provider fetches.

Shared registrations

  • Added tests/test_request_cancellation_worker.mjs to test:worker.
  • Added worker/upstream-cancellation.mjs to the secret firewall inventory.
  • tests/helpers/load_cloudflare_worker.mjs now resolves the new transport import.

Tests

  • tests/test_request_cancellation.py and tests/test_request_cancellation_worker.mjs: new.
  • Full Python suite: 3,193 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,239 passed, including the existing roleplay transport suites. cf:migrations:check, test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

This is part of Wave 3. It is based on the Wave 2 integration branch (#25), and the Wave 3 integration PR adds the README line.

Give Flask streaming iterators and the Worker roleplay transport a single cancellation owner, so a disconnect, read error or completion releases the upstream exactly once and is recorded as an ambiguous outcome with unknown usage. A failed SSE socket read now ends the stream instead of re-reading the partly consumed response.
@coderabbitai

coderabbitai Bot commented Oct 8, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e044d358-a235-4661-ae01-d179fc7c4a3c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-08T22:02:01.048957Z 6d34713 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
multillm-proxy 6d34713 Oct 08 2026, 09:59 PM

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6d34713a77

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +138 to +142
response.close = context.close
if isinstance(response, Response):
source = response.response
response.response = CancellationIterator(
lambda: iter(source), context, close_source=getattr(source, "close", None),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Mark fully consumed raw responses as complete

For unified requests returning a requests.Response, this replaces close() but only wraps Flask Response bodies. The raw passthrough path in stream_upstream_response() therefore consumes the complete body and invokes this overridden close() from its finally, which records cancelled with ambiguous=true instead of completion. Normal raw chat, image, and protocol responses will consequently expose an uncertain cancellation outcome even when every byte was delivered; the raw-body iterator needs to signal complete() on EOF before cleanup.

Useful? React with 👍 / 👎.

Comment on lines +57 to +58
def complete(self) -> None:
self._finish("complete", classify_upstream_outcome(200))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Classify completion using the actual response status

Every exhausted response is classified as HTTP 200 here, regardless of its real status. For example, fully consuming an owned 401, 429, or 503 response produces an UpstreamOutcome reporting provider success and accepted credentials, contradicting the repository's status classification and potentially corrupting health or credential accounting once this callback is consumed. Capture the response status when binding and classify completion with that value.

Useful? React with 👍 / 👎.

Comment on lines +58 to +60
close(reason = "cancelled") {
if (this.settled) return Promise.resolve();
this.finish(reason === "upstream_header_timeout" ? reason : "cancelled");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve interrupted outcomes when closing Worker streams

When an upstream body read throws, wrapResponse() explicitly calls close("interrupted"), but this expression rewrites every reason except the header timeout to cancelled. The same misclassification occurs for transport failures cleaned up after fetch rejection, so observers cannot distinguish provider/socket interruption from a client cancellation even though reason is part of the new outcome contract. Preserve interrupted rather than collapsing it to cancelled.

Useful? React with 👍 / 👎.

return Promise.resolve();
}

cleanup() { void this.close(); }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep cleanup compatible with subsequent body cancellation

The new cleanup() now errors the wrapped downstream stream via close(), so callers that intentionally discard a response by calling cleanup() and then awaiting response.body.cancel() receive a rejected AbortError. This affects the invalid-content-type branches in worker/roleplay/continuation.mjs and worker/roleplay/nonstream-repair.mjs: instead of returning null and handling an unusable continuation, they propagate an exception and can fail the entire roleplay response. Make cleanup leave an explicitly discarded body cancellable, or update those cleanup sequences so intentional disposal cannot reject.

Useful? React with 👍 / 👎.

Comment on lines +130 to +132
context = getattr(response, "multillm_cancellation", None)
if isinstance(context, CancellationContext):
return context

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Attach observers when rebinding an owned response

If a response already has a cancellation context, this returns it without applying the newly supplied on_outcome. Unified dispatch now binds every response without an observer, so a later accounting or settlement layer calling the documented bind_cancellation(response, on_outcome=callback) API will silently receive the existing context and its callback will never run. Preserve or register the new observer when reusing a context, or arrange for the initial unified binding to receive it.

Useful? React with 👍 / 👎.

@ALikesToCode
ALikesToCode merged commit 9e456fc into integrate/gateway-waves-1-2 Oct 9, 2026
5 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the feat/request-cancellation branch October 26, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant